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IN THE CLAIMS 

1. (Currently Amended) A computer readable storage m edium having 
instructions, which when implemented by a computer generate client 
side markup for a client in a server/client system, the 
instructions comprising: 

a set of modality dependent controls defined in an authoring 
page for a website for defining desired visual 
renderings and at least one of recognition and audible 
prompting on a client in a server/client system, each 
control having a first set of attributes directly 
related to visual rendering and a second set of 
attributes directly related to at least one of 
recognition and audibly prompting, the controls being 
related to client side markup executable by a client 
browser; and 

a module operable on a computer, the module being configured 
to receive the authoring page, and wherein the module 
is further configured to generate, using modality 
dependent attributes provided directly from the 
controls on the authoring page, client side markup 
executable by the client browser on the client in the 
server/ client system in accordance with the controls 
and the attributes of the controls to perform both 
visual rendering and at least one of recognition and 
audibly prompting. 

2. (Currently Amended) The computer readable storage m edium of 
claim 1 wherein one of the second set of attributes for one of the 
controls relates to a grammar to use for recognition. 



3. (Currently Amended) The computer readable storage m edium of 
claim 2 wherein said one of the second set of attributes provides 
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a reference to a location of the grammar. 

4. (Currently Amended) The computer readable storage m edium of 
claim 2 wherein the grammar is for one of speech recognition, 
handwriting recognition, gesture recognition and visual 
recognition. 

5. (Currently Amended) The computer readable storage m edium of 
claim 4 wherein the controls relate to one of HTML, XHTML, cHTML, 
XML and WML. 

6. (Currently Amended) The computer readable storage m edium of 
claim 1 wherein the controls relate to one of HTML, XHTML, cHTML, 
XML and WML. 

7. (Currently Amended) The computer readable storage m edium of 
claim 1 wherein one of the second set of attributes for one of the 
controls provides instructions related to generating audible 
output . 

8. (Currently Amended) The computer readable storage m edium of 
claim 7 wherein the instructions comprise text and the attribute 
relates to converting the text to audible output. 

9. (Currently Amended) The computer readable storage m edium of 
claim 1 wherein one of the second set of attributes for one of the 
controls relates to a location of data for audible output. 

10. (Currently Amended) The computer readable storage m edium of 
claim 9 wherein the data comprises a prerecorded audio data file 
and the attribute relates to playing the prerecorded audio data 
file. 
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11. (Currently Amended) The computer readable storage medium of 
claim 9 wherein the data comprises text and the attribute relates 
to converting the text to audible output. 

12 . (Currently Amended) A computer readable storage medium having 
instructions defined on an authoring page for a website, which 
when implemented by a computer generate client side markup for a 
client in a server/client system, the instructions comprising: 

a first set of modality dependent visual controls 

having attributes directly related to defining 
desired visual renderings on the client device, 
the first set of controls being related to client 
side markup executable by a client browser, and 
wherein the attributes include at leapt one of 
location for vioual rendering, font, background 
color, and foreground color; and 

a second set of modality dependent controls having 

attributes directly related to defining desired 
operation on the client device comprising at least 
one of recognition and audibly prompting, wherein 
attributes related to audible prompting include at 
least one of inline text for text-to-speech 
conversion, location of data for audible rendering 
and playing of a prerecorded audio file, - the 
second set of controls using at least one of the 
first controls, the second set of controls being 
related to client side markup executable by a 
client browser; and 

a module operable on a computer, the module being 

configured to receive the authoring page, which 
includes a plurality of the second set of 
controls, wherein the module is further configured 
to process the plurality of the second set of 
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controls from the authoring page to generate 
client side markup directly from the modality 
dependent controls that is executable by the 
client browser on the client in the server/client 
system in accordance with the second set of 
controls and the attributes of the second set of 
controls for at least one of recognition and 
audibly prompting, and wherein the module is 
configured to use at least one of the first set of 
controls from the authoring page in order to 
generate markup directly therefrom when processing 
each of the second set of controls. 

13. (Currently Amended) The computer readable storage m edium of 
claim 12 wherein one of the attributes for the second set of 
controls relates to a grammar to use for recognition. 

14. (Currently Amended) The computer readable storage m edium of 
claim 13 wherein said one of the attributes provides a reference 
to a location of the grammar. 

15. (Currently Amended) The computer readable storage medium of 
claim 13 wherein the grammar is for one of speech recognition, 
handwriting recognition, gesture recognition and visual 
recognition. 

16. (Currently Amended) The computer readable storage medium of 
claim 15 wherein the first set of controls and the second set of 
controls relate to one of HTML, XHTML, cHTML, XML and WML. 

17. (Currently Amended) The computer readable storage medium of 
claim 12 wherein the controls relate to one of HTML, XHTML, cHTML, 
XML and WML. 
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18. (Currently Amended) The computer readable storage medium of 
claim 12 wherein one of the second set of attributes for one of 
the controls provides instructions related to generating audible 
output . 

19. (Currently Amended) The computer readable storage m edium of 
claim 18 wherein the instructions comprise text and the attribute 
relates to converting the text to audible output. 

20. (Currently Amended) The computer readable storage m edium of 
claim 12 wherein one of the second set of attributes for one of 
the controls relates to a location of data for audible output. 

21. (Currently Amended) The computer readable storage m edium of 
claim 20 wherein the data comprises a prerecorded audio data file 
and the attribute relates to playing the prerecorded audio data 
file. 

22. (Currently Amended) The computer readable storage m edium of 
claim 20 wherein the data comprises text and the attribute relates 
to converting the text to audible output. 

23. (Currently Amended) A computer readable storage m edium having 
instructions, which when implemented by a computer generate client 
side markup for a client in a server/client system, the 
instructions comprising: 

a first set of modality dependent v isual controls 

defined on an authoring page for a website having 
attributes directly related to a first modality of 
interaction with a user of the client that being 
defining dcaircd v isual renderings on the client 
device, the first set of controls being related to 
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client side markup executable by a client browser; 
and 

a second set of modality dependent controls defined on 
the authoring page for defining desired operation 
on the client device having attributes directly 
related to a second modality of interaction with a 
user of the client that being at least one of 
recognition and audible prompting, wherein 
attributes directly related to recognition include 
at least one of location of grammar for use in 
recognition and confidence level thresholds for 
recognition and wherein attributes directly 
related to audible prompting include at least one 
of inline text for text -to -speech conversion, 
location of data for audible rendering and playing 
of a prerecorded audio file, the second set of 
controls being selectively associated with the 
first set of controls, and the second set of 
controls being related to client side markup 
executable a client browser; and 

a module operable on a computer, the module being 

configured to receive the authoring page, wherein 
the module is further configured to process the 
controls of the first set in the authoring page to 
generate client side markup for the first modality 
of interaction by incorporating the attributes in 
the modality dependent controls that is executable 
by the client browser on the client in the 
server/client system in accordance with the 
controls of the first set and the attributes of 
the controls of the first set to perform both 
visual rendering, and wherein the module is 
further configured to process the controls of the 
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second set to generate client side markup for the 
second modality of interaction by incorporating 
the attributes in the modality dependent controls 
that is executable by the client browser on the 
client in the server/client system in accordance 
with the controls of the second set and the 
attributes of the controls of the second set in 
the authoring page to perform at least one of 
recognition and audibly prompting. 

24. (Currently Amended) The computer readable storage m edium of 
claim 23 wherein one of the attributes for the second set of 
controls relates to a grammar to use for recognition. 

25. (Currently Amended) The computer readable storage m edium of 
claim 24 wherein said one of the attributes provides a reference 
to a location of the grammar. 

26. (Currently Amended) The computer readable storage m edium of 
claim 24 wherein the grammar is for one of speech recognition, 
handwriting recognition, gesture recognition and visual 
recognition. 

27. (Currently Amended) The computer readable storage m edium of 
claim 26 wherein the first set of controls and the second set of 
controls relate to one of HTML, XHTML, cHTML , XML and WML. 

28. (Currently Amended) The computer readable storage m edium of 
claim 23 wherein the controls relate to one of HTML, XHTML, cHTML, 
XML and WML. 

29. (Currently Amended) The computer readable storage m edium of 
claim 23 wherein one of the second set of attributes for one of 
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the controls provides instructions related to generating audible 
output . 

30. (Currently Amended) The computer readable storage m edium of 
claim 29 wherein the instructions comprise text and the attribute 
relates to converting the text to audible output. 

31. (Currently Amended) The computer readable storage m edium of 
claim 23 wherein one of the second set of attributes for one of 
the controls relates to a location of data for audible output. 

32. (Currently Amended) The computer readable storage m edium of 
claim 31 wherein the data comprises a prerecorded audio data file 
and the attribute relates to playing the prerecorded audio data 
file. 

33. (Currently Amended) The computer readable storage m edium of 
claim 31 wherein the data comprises text and the attribute relates 
to converting the text to audible output. 

34. (Currently Amended) The computer readable storage medium of 
claim 23 wherein one of the attributes of the second set of 
controls relates to an identifier of the associated control of the 
first set of controls to form the association. 

35. (Currently Amended) The computer readable storage m edium of 
claim 23 wherein one of the attributes relates to whether the 
associated control of the second set is available for activation. 

I 36. (Currently Amended) The computer readable storage m edium of 
claim 35 wherein activation relates to generating markup. 

I 37. (Currently Amended) The computer readable storage m edium of 
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claim 35 wherein activation relates to execution on the client. 

38. (Currently Amended) The computer readable storage m edium of 
claim 23 wherein the second set of controls activates another 
control of the second set. 

39. (Currently Amended) The computer readable storage medium of 
claim 23 wherein the second set of controls comprise: 

a question control for generating markup related to audible 

prompting of a question; and 
an answer control for generating markup related to a grammar 
for recognition. 

40. (Currently Amended) The computer readable storage m edium of 
claim 39 wherein the question control activates the answer 
control . 

41. (Currently Amended) The computer readable storage m edium of 
claim 40 wherein the answer control includes a mechanism to 
associate a received result with one of the first set of controls. 

42. (Currently Amended) The computer readable storage m edium of 
claim 41 wherein mechanism includes binding the recognition value. 

43. (Previously Presented) The computer readable storage medium 
of claim 42 wherein the mechanism includes issuing an event 
related to operation of binding. 

| 44. (Currently Amended) The computer readable storage m edium of 
claim 40 wherein the second set of controls comprise: 

a command control for generating markup related to a 

grammar for one of navigation in the markup, help 
with a task, and repeating an audible prompt. 
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45. (Currently Amended) The computer readable storage m edium of 
claim 40 wherein the second set of controls comprise: 

a confirmation control for generating markup related to 
confirming that a recognized result is correct. 

46. (Currently Amended) The computer readable storage m edium of 
claim 45 wherein the confirmation control includes an attribute 
related to the recognized result to be confirmed. 

47. (Currently Amended) The computer readable storage m edium of 
claim 46 wherein the answer control includes an attribute related 
to a confidence level . 

48. (Currently Amended) The computer readable storage medium of 
claim 46 wherein the confirmation control is activated as a 
function of a confidence level of a received result. 

49. (Currently Amended) The computer readable storage m edium of 
claim 48 wherein the confirmation control activates an accept 
control to accept the recognized result. 

50. (Currently Amended) The computer readable storage m edium of 
claim 48 wherein the confirmation control activates a deny control 
to deny the recognized result. 

51. (Currently Amended) The computer readable storage m edium of 
claim 48 wherein the confirmation control activates a correct 
control to correct the recognized result. 

52 . (Currently Amended) A computer implemented method for 
defining a website application on a server in a server/client 
architecture, the website application providing markup to a client 
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for performing recognition and/or audible prompting on the client, 
the method comprising: 

defining the website application by creating an 
authoring page with a first set of modality 
dependent v isual controls having attributes 
directly related to a first modality of 
interaction with a user of the client that being 
visual rendering on the client device, fefee 
attributes directly related to visual rendering 
including at lcaat one of location for visual 
rendering, font, background color, and foreground 
color and with a second set of modality dependent 
controls having attributes directly related to a 
second modality of interaction with the user of 
the client that being at least one of recognition 
and audibly prompting, wherein attributes related 
to audible prompting include at least one of 
inline text for text- to- speech conversion, 
location of data for audible rendering and playing 
of a prerecorded audio file, and wherein the first 
set of controls and the second set of controls 
being related to client side markup executable by 
a client browser, wherein defining includes 
selectively associating controls of the second set 
of controls with at least one control of the first 
set of visual controls; and 
processing the controls on the authoring page to 

directly generate client side markup including 
processing of the first set of visual controls to 
generate client side markup for the first modality 
of interaction executable by the client browser on 
the client in the server/client system in 
accordance with the modality dependent controls of 
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the first set and the attributes of the controls 
of the first set to perform both v isual rendering, 
and processing the controls of the second set to 
generate client side markup for the second 
modality of interaction b y incorporating the 
attributes in the modality dependent controls that 
is executable by the client browser on the client 
in the Dcrvcr/olicnt system in accordance with the 
controls of the second set and the attributes of 
the controls of the second act to perform at least 
one of recognition and audibly prompting. 

53. (Withdrawn) The computer implemented method of claim 52 
wherein each of the controls of the second set include an 
identifier attribute for identifying a control of the first set of 
visual controls, and wherein associating includes providing an 
identifier of at least one control of the first set of controls in 
the corresponding identifier attribute of each of the second set 
of controls. 

54. (Withdrawn) The computer implemented method of claim 52 
wherein the second set of controls includes a question control 
related to audible prompting of a question, and an answer control 
related to a grammar for recognition; and wherein defining the 
website application with a second set of controls related to at 
least one of recognition and audibly prompting includes 
associating the answer control with the question control. 

55. (Withdrawn) The computer implemented method of claim 54 
wherein the second set of controls includes a confirmation control 
related to confirming that a recognized result is correct; and 
wherein defining the website application with a second set of 
controls related to at least one of recognition and audibly 
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prompting includes associating the confirmation control with a 
recognized result to be received. 

56. (Withdrawn) The computer implemented method of claim 55 
wherein the second set of controls includes a command control for 
generating markup related to a grammar for one of navigation on 
the computer, help with a task, and repeating an audible prompt; 
and wherein defining the website application with a second set of 
controls related to at least one of recognition and audibly 
prompting includes associating the command control with a question 
control . 



